{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO",
    "tags": []
   },
   "source": [
    "# Creating a Vertex AI Feature Store Using Pandas\n",
    "\n",
    "### Learning objectives\n",
    "\n",
    "1. Create Feature Store Resources.\n",
    "2. Ingest Feature Values into Entity Type from a Pandas DataFrame.\n",
    "3. Read/Online Serve Entity's Feature Values from Vertex AI Online Feature Store.\n",
    "4. Batch Serve Feature Values from Vertex AI Feature Store.\n",
    "5. Read the Updated Feature Values.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c4ZNLaf6T0lN"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "This notebook introduces Pandas support for Feature Store using Vertex AI SDK. For pre-requisites and introduction on Vertex AI SDK and Feature Store native support, please go through this [Colab notebook](https://colab.sandbox.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/sdk-feature-store.ipynb).\n",
    "\n",
    "### Dataset\n",
    "\n",
    "This notebook uses a movie recommendation dataset as an example throughout all the notebooks including this one. The original task is to train a model to predict if a user is going to watch a movie and serve the model online.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/sdk-feature-store-pandas.ipynb) -- try to complete that notebook first before reviewing this solution notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lWEdiXsJg0XY"
   },
   "source": [
    "## Before you begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "### Install additional packages\n",
    "\n",
    "To run this notebook, you need to install the following packages for Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m  WARNING: The script avro is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script tb-gcp-uploader is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "google-cloud-recommendations-ai 0.2.0 requires google-api-core[grpc]<2.0.0dev,>=1.22.2, but you have google-api-core 2.8.1 which is incompatible.\n",
      "apache-beam 2.40.0 requires dill<0.3.2,>=0.3.1.1, but you have dill 0.3.5.1 which is incompatible.\n",
      "apache-beam 2.40.0 requires pyarrow<8.0.0,>=0.15.1, but you have pyarrow 9.0.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\"\n",
    "    \n",
    "! pip install -U {USER_FLAG} --upgrade google-cloud-aiplatform \\\n",
    "                                        google-cloud-bigquery \\\n",
    "                                        google-cloud-bigquery-storage \\\n",
    "                                        avro \\\n",
    "                                        pyarrow \\\n",
    "                                        pandas -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhq5zEbGg0XX"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the packages, you need to restart the notebook kernel so that it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EzrelQZ22IZj"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs\n",
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WReHDGG5g0XY"
   },
   "source": [
    "#### Set your project ID\n",
    "\n",
    "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "dcdfccf50581"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"[your-project-id]\"  # Replace with your project ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oM1iC_MfAts1"
   },
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
    "    # Get your GCP project id from gcloud\n",
    "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID:\", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "09021c90b34c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f41eda68c379"
   },
   "source": [
    "#### Region\n",
    "\n",
    "You can also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
    "\n",
    "- Americas: `us-central1`\n",
    "- Europe: `europe-west4`\n",
    "- Asia Pacific: `asia-east1`\n",
    "\n",
    "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
    "\n",
    "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "5c615e53149f"
   },
   "outputs": [],
   "source": [
    "REGION = \"[your-region]\"  # Replace with your region\n",
    "\n",
    "if REGION == \"[your-region]\":\n",
    "    REGION = \"us-central1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XoEqT2Y4DJmf"
   },
   "source": [
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "Cdct_Lm7x2I_"
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "import pandas as pd\n",
    "from google.cloud import aiplatform\n",
    "\n",
    "aiplatform.init(project=PROJECT_ID, location=REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9UvxYyGUimKw"
   },
   "source": [
    "## Create Feature Store Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "buQBIv3ZL3A0"
   },
   "source": [
    "### Create Feature Store\n",
    "\n",
    "The method to create a Feature Store returns a\n",
    "[long-running operation](https://google.aip.dev/151) (LRO). An LRO starts an asynchronous job. LROs are returned for other API\n",
    "methods too, such as updating or deleting a featurestore. Running the code cell creates a featurestore and prints the process logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "D6uIWQeoBSr8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Featurestore\n",
      "Create Featurestore backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/operations/5097191114432053248\n",
      "Featurestore created. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions\n",
      "To use this Featurestore in another session:\n",
      "featurestore = aiplatform.Featurestore('projects/816732343711/locations/us-central1/featurestores/movie_predictions')\n"
     ]
    }
   ],
   "source": [
    "movie_predictions_feature_store = aiplatform.Featurestore.create(\n",
    "    featurestore_id=\"movie_predictions\",\n",
    "    online_store_fixed_node_count=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EpmJq75zXjmT"
   },
   "source": [
    "### Create Entity Types\n",
    "\n",
    "Entity types can be created within the Featurestore class. Below, you create the `Users` entity type and `Movies` entity type. Process logs are printed in the output for each cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "GU0oXvINBgPV"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating EntityType\n",
      "Create EntityType backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/operations/1886124580116889600\n",
      "EntityType created. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users\n",
      "To use this EntityType in another session:\n",
      "entity_type = aiplatform.EntityType('projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users')\n"
     ]
    }
   ],
   "source": [
    "# TODO 1\n",
    "# Create the Users entity type\n",
    "users_entity_type = movie_predictions_feature_store.create_entity_type(\n",
    "    entity_type_id=\"users\",\n",
    "    description=\"Users entity\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "qPCGFznrFwFy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating EntityType\n",
      "Create EntityType backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies/operations/7891111733262024704\n",
      "EntityType created. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n",
      "To use this EntityType in another session:\n",
      "entity_type = aiplatform.EntityType('projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies')\n"
     ]
    }
   ],
   "source": [
    "# TODO 2\n",
    "# Create the Movies entity type\n",
    "movies_entity_type = movie_predictions_feature_store.create_entity_type(\n",
    "    entity_type_id=\"movies\",\n",
    "    description=\"Movies entity\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FJW4q-0jO2Xf"
   },
   "source": [
    "### Create Features\n",
    "Features can be created within each entity type. Add defining features to the `Users` entity type and `Movies` entity type by using the following methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "PvjwT84iVSps"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Feature\n",
      "Create Feature backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/operations/5925853445868224512\n",
      "Feature created. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/features/age\n",
      "To use this Feature in another session:\n",
      "feature = aiplatform.Feature('projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/features/age')\n",
      "Creating Feature\n",
      "Create Feature backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/operations/897021511955644416\n",
      "Feature created. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/features/gender\n",
      "To use this Feature in another session:\n",
      "feature = aiplatform.Feature('projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/features/gender')\n",
      "Creating Feature\n",
      "Create Feature backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/operations/3977483657077063680\n",
      "Feature created. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/features/liked_genres\n",
      "To use this Feature in another session:\n",
      "feature = aiplatform.Feature('projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/features/liked_genres')\n"
     ]
    }
   ],
   "source": [
    "users_feature_age = users_entity_type.create_feature(\n",
    "    feature_id=\"age\",\n",
    "    value_type=\"INT64\",\n",
    "    description=\"User age\",\n",
    ")\n",
    "\n",
    "users_feature_gender = users_entity_type.create_feature(\n",
    "    feature_id=\"gender\",\n",
    "    value_type=\"STRING\",\n",
    "    description=\"User gender\",\n",
    ")\n",
    "\n",
    "users_feature_liked_genres = users_entity_type.create_feature(\n",
    "    feature_id=\"liked_genres\",\n",
    "    value_type=\"STRING_ARRAY\",\n",
    "    description=\"An array of genres this user liked\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "llTT9_Dgbac2"
   },
   "outputs": [],
   "source": [
    "movies_feature_configs = {\n",
    "    \"title\": {\n",
    "        \"value_type\": \"STRING\",\n",
    "        \"description\": \"The title of the movie\",\n",
    "    },\n",
    "    \"genres\": {\n",
    "        \"value_type\": \"STRING\",\n",
    "        \"description\": \"The genre of the movie\",\n",
    "    },\n",
    "    \"average_rating\": {\n",
    "        \"value_type\": \"DOUBLE\",\n",
    "        \"description\": \"The average rating for the movie, range is [1.0-5.0]\",\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "YhfOKJL_BvuM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch creating features EntityType entityType: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n",
      "Batch create Features EntityType entityType backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/operations/8192852908295847936\n",
      "EntityType entityType Batch created features. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n"
     ]
    }
   ],
   "source": [
    "movie_features = movies_entity_type.batch_create_features(\n",
    "    feature_configs=movies_feature_configs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K3n5XdK8Xjmw"
   },
   "source": [
    "## Ingest Feature Values into Entity Type from a Pandas DataFrame\n",
    "\n",
    "You need to ingest feature values into your entity type containing the features, so you can later `read` (online) or `batch serve` (offline) the feature values from the entity type. In this step, you will learn how to ingest feature values from a Pandas DataFrame into an entity type. We can also import feature values from BigQuery or Google Cloud Storage.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BlqJ-QdTcs6W"
   },
   "source": [
    "#### Get data from source files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "_uNrHqiGXrff"
   },
   "outputs": [],
   "source": [
    "GCS_USERS_AVRO_URI = (\n",
    "    \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/users.avro\"\n",
    ")\n",
    "GCS_MOVIES_AVRO_URI = (\n",
    "    \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movies.avro\"\n",
    ")\n",
    "\n",
    "USERS_AVRO_FN = \"users.avro\"\n",
    "MOVIES_AVRO_FN = \"movies.avro\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "KqIH_bS-5OW5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/users.avro...\n",
      "/ [1 files][  637.0 B/  637.0 B]                                                \n",
      "Operation completed over 1 objects/637.0 B.                                      \n",
      "Copying gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movies.avro...\n",
      "/ [1 files][  599.0 B/  599.0 B]                                                \n",
      "Operation completed over 1 objects/599.0 B.                                      \n"
     ]
    }
   ],
   "source": [
    "! gcloud storage cp $GCS_USERS_AVRO_URI $USERS_AVRO_FN\n",    "! gcloud storage cp $GCS_MOVIES_AVRO_URI $MOVIES_AVRO_FN"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fd6Z0jfR5OW5"
   },
   "source": [
    "#### Load Avro Files into Pandas DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "KrB7bnqbZYaC"
   },
   "outputs": [],
   "source": [
    "from avro.datafile import DataFileReader\n",
    "from avro.io import DatumReader\n",
    "\n",
    "\n",
    "class AvroReader:\n",
    "    def __init__(self, data_file):\n",
    "        self.avro_reader = DataFileReader(open(data_file, \"rb\"), DatumReader())\n",
    "\n",
    "    def to_dataframe(self):\n",
    "        records = [record for record in self.avro_reader]\n",
    "        return pd.DataFrame.from_records(data=records)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "XdlWJhUt5OW5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   user_id   age  gender               liked_genres  \\\n",
      "0     dave   NaN    None  [Children's, Documentary]   \n",
      "1    alice   NaN    None            [Drama, Comedy]   \n",
      "2  charlie   NaN    None        [Sci-Fi, Animation]   \n",
      "3      bob   NaN    None            [Action, Crime]   \n",
      "4      eve  26.0    None                   [Horror]   \n",
      "5      bob  35.0    Male        [Action, Adventure]   \n",
      "6    alice  55.0  Female                    [Drama]   \n",
      "\n",
      "                       update_time  \n",
      "0 2021-08-20 20:58:22.261581+00:00  \n",
      "1 2021-08-20 20:58:22.261581+00:00  \n",
      "2 2021-08-20 20:58:22.261581+00:00  \n",
      "3 2021-08-20 20:58:22.261581+00:00  \n",
      "4 2021-08-20 20:58:22.261581+00:00  \n",
      "5 2021-08-20 20:58:22.261581+00:00  \n",
      "6 2021-08-20 20:58:22.261581+00:00  \n"
     ]
    }
   ],
   "source": [
    "users_avro_reader = AvroReader(data_file=USERS_AVRO_FN)\n",
    "users_source_df = users_avro_reader.to_dataframe()\n",
    "print(users_source_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "gZ49cPS35OW5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   movie_id  average_rating                     title   genres  \\\n",
      "0  movie_02             4.2               The Shining   Horror   \n",
      "1  movie_04             4.6           The Dark Knight   Action   \n",
      "2  movie_03             4.5           Cinema Paradiso  Romance   \n",
      "3  movie_01             4.9  The Shawshank Redemption    Drama   \n",
      "\n",
      "                       update_time  \n",
      "0 2021-08-20 20:44:11.094375+00:00  \n",
      "1 2021-08-20 20:44:11.094375+00:00  \n",
      "2 2021-08-20 20:44:11.094375+00:00  \n",
      "3 2021-08-20 20:44:11.094375+00:00  \n"
     ]
    }
   ],
   "source": [
    "movies_avro_reader = AvroReader(data_file=MOVIES_AVRO_FN)\n",
    "movies_source_df = movies_avro_reader.to_dataframe()\n",
    "print(movies_source_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bgb0WGwX5OW6"
   },
   "source": [
    "#### Ingest Feature Values into _Users_ Entity Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "76b813uj5OW6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing EntityType feature values: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users\n",
      "Import EntityType feature values backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/operations/1653626249353887744\n",
      "EntityType feature values imported. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f2afb858450> \n",
       "resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO 3\n",
    "# Ingest Feature Values into Users Entity Type\n",
    "users_entity_type.ingest_from_df(\n",
    "    feature_ids=[\"age\", \"gender\", \"liked_genres\"],\n",
    "    feature_time=\"update_time\",\n",
    "    df_source=users_source_df,\n",
    "    entity_id_field=\"user_id\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PCAdQ3cF5OW6"
   },
   "source": [
    "#### Ingest Feature Values into _Movies_ Entity Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "-DYlKe4e5OW6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing EntityType feature values: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n",
      "Import EntityType feature values backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies/operations/1220717735172898816\n",
      "EntityType feature values imported. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f2afb6dd750> \n",
       "resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO 4\n",
    "# Ingest Feature Values into Movies Entity Type\n",
    "movies_entity_type.ingest_from_df(\n",
    "    feature_ids=[\"average_rating\", \"title\", \"genres\"],\n",
    "    feature_time=\"update_time\",\n",
    "    df_source=movies_source_df,\n",
    "    entity_id_field=\"movie_id\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pIYLZwao5OW6"
   },
   "source": [
    "## Read/Online Serve Entity's Feature Values from Vertex AI Online Feature Store\n",
    "\n",
    "Feature Store allows [online serving](https://cloud.google.com/vertex-ai/docs/featurestore/serving-online)\n",
    "which lets you read feature values for small batches of entities. It works well when you want to read values of selected features from an entity or multiple entities in an entity type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "qrR-SY3i58rh"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  entity_id   age               liked_genres  gender\n",
      "0     alice  55.0                    [Drama]  Female\n",
      "1       bob  35.0        [Action, Adventure]    Male\n",
      "2   charlie   NaN        [Sci-Fi, Animation]    None\n",
      "3      dave   NaN  [Children's, Documentary]    None\n",
      "4       eve  26.0                   [Horror]    None\n"
     ]
    }
   ],
   "source": [
    "users_read_df = users_entity_type.read(\n",
    "    entity_ids=[\"dave\", \"alice\", \"charlie\", \"bob\", \"eve\"],\n",
    ")\n",
    "print(users_read_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "vTW6kBxN5OW7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  entity_id                     title   genres  average_rating\n",
      "0  movie_01  The Shawshank Redemption    Drama             4.9\n",
      "1  movie_02               The Shining   Horror             4.2\n",
      "2  movie_03           Cinema Paradiso  Romance             4.5\n",
      "3  movie_04           The Dark Knight   Action             4.6\n"
     ]
    }
   ],
   "source": [
    "movies_read_df = movies_entity_type.read(\n",
    "    entity_ids=[\"movie_01\", \"movie_02\", \"movie_03\", \"movie_04\"],\n",
    "    feature_ids=[\"title\", \"genres\", \"average_rating\"],\n",
    ")\n",
    "print(movies_read_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AK2Glzkq5OW7"
   },
   "source": [
    "## Batch Serve Feature Values from Vertex AI Feature Store\n",
    "\n",
    "Batch Serving is used to fetch a large batch of feature values for high-throughput, and is typically used for training a model or batch prediction. In this section, you learn how to prepare training examples by using the Feature Store's batch serve function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hxsotHUe5OW7"
   },
   "source": [
    "#### Read instances from source file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "G4k2QVN-5OW7"
   },
   "outputs": [],
   "source": [
    "GCS_READ_INSTANCES_CSV_URI = \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movie_prediction.csv\"\n",
    "READ_INSTANCES_CSV_FN = \"data.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "Rr8XDjk_5OW7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movie_prediction.csv...\n",
      "/ [1 files][  255.0 B/  255.0 B]                                                \n",
      "Operation completed over 1 objects/255.0 B.                                      \n"
     ]
    }
   ],
   "source": [
    "! gcloud storage cp $GCS_READ_INSTANCES_CSV_URI $READ_INSTANCES_CSV_FN"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T5DW1MFt5OW7"
   },
   "source": [
    "#### Load CSV file into a Pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "JqQVfRnC5OW7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   users    movies             timestamp\n",
      "0  alice  movie_01  2021-09-15T08:28:14Z\n",
      "1    bob  movie_02  2021-09-15T08:28:14Z\n",
      "2    dav  movie_03  2021-09-15T08:28:14Z\n",
      "3    eve  movie_04  2021-09-15T08:28:14Z\n",
      "4  alice  movie_03  2021-09-14T09:35:15Z\n",
      "5    bob  movie_04  2020-02-14T09:35:15Z\n"
     ]
    }
   ],
   "source": [
    "# TODO 5\n",
    "read_instances_df = pd.read_csv(READ_INSTANCES_CSV_FN)\n",
    "print(read_instances_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LsgNNH8G5OW8"
   },
   "source": [
    "#### Change the Dtype of `Timestamp` to `Datetime64`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "vb9ntNEA5OW8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before:  object\n",
      "after:   datetime64[ns]\n"
     ]
    }
   ],
   "source": [
    "print(\"before: \", read_instances_df[\"timestamp\"].dtype)\n",
    "read_instances_df = read_instances_df.astype({\"timestamp\": \"datetime64\"})\n",
    "print(\"after:  \", read_instances_df[\"timestamp\"].dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ao1dC5Pc5OW8"
   },
   "source": [
    "#### Batch Serve Feature Values from Movie Predictions Feature Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "vZSJ-Sbl5OW8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Serving Featurestore feature values: projects/816732343711/locations/us-central1/featurestores/movie_predictions\n",
      "Serve Featurestore feature values backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/operations/9130868268184109056\n",
      "Featurestore feature values served. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>entity_type_users</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>liked_genres</th>\n",
       "      <th>entity_type_movies</th>\n",
       "      <th>title</th>\n",
       "      <th>average_rating</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-02-14 09:35:15+00:00</td>\n",
       "      <td>bob</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>movie_04</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-09-15 08:28:14+00:00</td>\n",
       "      <td>dav</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>movie_03</td>\n",
       "      <td>Cinema Paradiso</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-09-15 08:28:14+00:00</td>\n",
       "      <td>eve</td>\n",
       "      <td>26.0</td>\n",
       "      <td>None</td>\n",
       "      <td>[Horror]</td>\n",
       "      <td>movie_04</td>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>4.6</td>\n",
       "      <td>Action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-09-15 08:28:14+00:00</td>\n",
       "      <td>bob</td>\n",
       "      <td>35.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>[Action, Crime]</td>\n",
       "      <td>movie_02</td>\n",
       "      <td>The Shining</td>\n",
       "      <td>4.2</td>\n",
       "      <td>Horror</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-09-15 08:28:14+00:00</td>\n",
       "      <td>alice</td>\n",
       "      <td>55.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>[Drama]</td>\n",
       "      <td>movie_01</td>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>4.9</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2021-09-14 09:35:15+00:00</td>\n",
       "      <td>alice</td>\n",
       "      <td>55.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>[Drama]</td>\n",
       "      <td>movie_03</td>\n",
       "      <td>Cinema Paradiso</td>\n",
       "      <td>4.5</td>\n",
       "      <td>Romance</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  timestamp entity_type_users   age  gender     liked_genres  \\\n",
       "0 2020-02-14 09:35:15+00:00               bob   NaN    None               []   \n",
       "1 2021-09-15 08:28:14+00:00               dav   NaN    None               []   \n",
       "2 2021-09-15 08:28:14+00:00               eve  26.0    None         [Horror]   \n",
       "3 2021-09-15 08:28:14+00:00               bob  35.0    Male  [Action, Crime]   \n",
       "4 2021-09-15 08:28:14+00:00             alice  55.0  Female          [Drama]   \n",
       "5 2021-09-14 09:35:15+00:00             alice  55.0  Female          [Drama]   \n",
       "\n",
       "  entity_type_movies                     title  average_rating   genres  \n",
       "0           movie_04                      None             NaN     None  \n",
       "1           movie_03           Cinema Paradiso             4.5  Romance  \n",
       "2           movie_04           The Dark Knight             4.6   Action  \n",
       "3           movie_02               The Shining             4.2   Horror  \n",
       "4           movie_01  The Shawshank Redemption             4.9    Drama  \n",
       "5           movie_03           Cinema Paradiso             4.5  Romance  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_predictions_df = movie_predictions_feature_store.batch_serve_to_df(\n",
    "    serving_feature_ids={\n",
    "        \"users\": [\"age\", \"gender\", \"liked_genres\"],\n",
    "        \"movies\": [\"title\", \"average_rating\", \"genres\"],\n",
    "    },\n",
    "    read_instances_df=read_instances_df,\n",
    ")\n",
    "movie_predictions_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "29gLNORP5OW8"
   },
   "source": [
    "## Read the Updated Feature Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XN84znoI5OW8"
   },
   "source": [
    "#### Feature Values from last ingestion\n",
    "Recall read from the Entity Type shows Feature Values from the last ingestion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "wtmshq_n5OW9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  entity_id                     title   genres  average_rating\n",
      "0  movie_01  The Shawshank Redemption    Drama             4.9\n",
      "1  movie_02               The Shining   Horror             4.2\n",
      "2  movie_03           Cinema Paradiso  Romance             4.5\n",
      "3  movie_04           The Dark Knight   Action             4.6\n"
     ]
    }
   ],
   "source": [
    "print(movies_read_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "feTUJjqG5OW9"
   },
   "source": [
    "#### Ingest updated Feature Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "2Y-iMUFH5OW9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   movie_id  average_rating\n",
      "0  movie_03             4.3\n",
      "1  movie_04             4.8\n"
     ]
    }
   ],
   "source": [
    "update_movies_df = pd.DataFrame(\n",
    "    data=[[\"movie_03\", 4.3], [\"movie_04\", 4.8]],\n",
    "    columns=[\"movie_id\", \"average_rating\"],\n",
    ")\n",
    "print(update_movies_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "aKKhSzUc5OW9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing EntityType feature values: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n",
      "Import EntityType feature values backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies/operations/7615406993573937152\n",
      "EntityType feature values imported. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f2afb6dd750> \n",
       "resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_entity_type.ingest_from_df(\n",
    "    feature_ids=[\"average_rating\"],\n",
    "    feature_time=datetime.datetime.now(),\n",
    "    df_source=update_movies_df,\n",
    "    entity_id_field=\"movie_id\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s47WCIvL5OW9"
   },
   "source": [
    "#### Latest Feature Values\n",
    "Read from the Entity Type shows updated Feature values from the latest ingestion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "_2IPEY7S5OW9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  entity_id                     title   genres  average_rating\n",
      "0  movie_01  The Shawshank Redemption    Drama             4.9\n",
      "1  movie_02               The Shining   Horror             4.2\n",
      "2  movie_03           Cinema Paradiso  Romance             4.3\n",
      "3  movie_04           The Dark Knight   Action             4.8\n"
     ]
    }
   ],
   "source": [
    "update_movies_read_df = movies_entity_type.read(\n",
    "    entity_ids=[\"movie_01\", \"movie_02\", \"movie_03\", \"movie_04\"],\n",
    "    feature_ids=[\"title\", \"genres\", \"average_rating\"],\n",
    ")\n",
    "print(update_movies_read_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wsvCRzn_5OW9"
   },
   "source": [
    "## Point-in-Time Correctness"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "R1YGRNsW5OW9"
   },
   "source": [
    "#### Missing data\n",
    "Recall Batch Serve from the last ingestion has some missing data in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "ueXYomBr5OW-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                  timestamp entity_type_users   age  gender     liked_genres  \\\n",
      "0 2020-02-14 09:35:15+00:00               bob   NaN    None               []   \n",
      "1 2021-09-15 08:28:14+00:00               dav   NaN    None               []   \n",
      "2 2021-09-15 08:28:14+00:00               eve  26.0    None         [Horror]   \n",
      "3 2021-09-15 08:28:14+00:00               bob  35.0    Male  [Action, Crime]   \n",
      "4 2021-09-15 08:28:14+00:00             alice  55.0  Female          [Drama]   \n",
      "5 2021-09-14 09:35:15+00:00             alice  55.0  Female          [Drama]   \n",
      "\n",
      "  entity_type_movies                     title  average_rating   genres  \n",
      "0           movie_04                      None             NaN     None  \n",
      "1           movie_03           Cinema Paradiso             4.5  Romance  \n",
      "2           movie_04           The Dark Knight             4.6   Action  \n",
      "3           movie_02               The Shining             4.2   Horror  \n",
      "4           movie_01  The Shawshank Redemption             4.9    Drama  \n",
      "5           movie_03           Cinema Paradiso             4.5  Romance  \n"
     ]
    }
   ],
   "source": [
    "print(movie_predictions_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "abQRF6mx5OW-"
   },
   "source": [
    "#### Backfill/Correct point-in-time data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "ehZhc4ZP5OW-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  user_id  age gender liked_genres         update_time\n",
      "0     bob   34   Male      [Drama] 2020-02-13 09:35:15\n"
     ]
    }
   ],
   "source": [
    "backfill_users_df = pd.DataFrame(\n",
    "    data=[[\"bob\", 34, \"Male\", [\"Drama\"], \"2020-02-13 09:35:15\"]],\n",
    "    columns=[\"user_id\", \"age\", \"gender\", \"liked_genres\", \"update_time\"],\n",
    ")\n",
    "backfill_users_df = backfill_users_df.astype({\"update_time\": \"datetime64\"})\n",
    "print(backfill_users_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "mhdTzl5k5OW-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   movie_id  average_rating            title  genres         update_time\n",
      "0  movie_04             4.2  The Dark Knight  Action 2020-02-13 09:35:15\n"
     ]
    }
   ],
   "source": [
    "backfill_movies_df = pd.DataFrame(\n",
    "    data=[[\"movie_04\", 4.2, \"The Dark Knight\", \"Action\", \"2020-02-13 09:35:15\"]],\n",
    "    columns=[\"movie_id\", \"average_rating\", \"title\", \"genres\", \"update_time\"],\n",
    ")\n",
    "backfill_movies_df = backfill_movies_df.astype({\"update_time\": \"datetime64\"})\n",
    "print(backfill_movies_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WXb4JUhu5OW-"
   },
   "source": [
    "#### Ingest backfilled/corrected point-in-time data from dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "vM1ejZMa5OW-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing EntityType feature values: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users\n",
      "Import EntityType feature values backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users/operations/541237141393375232\n",
      "EntityType feature values imported. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f2afb858450> \n",
       "resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/users"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_entity_type.ingest_from_df(\n",
    "    feature_ids=[\"age\", \"gender\", \"liked_genres\"],\n",
    "    feature_time=\"update_time\",\n",
    "    df_source=backfill_users_df,\n",
    "    entity_id_field=\"user_id\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "lBnrNbv75OW-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing EntityType feature values: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n",
      "Import EntityType feature values backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies/operations/1467289814771433472\n",
      "EntityType feature values imported. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f2afb6dd750> \n",
       "resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions/entityTypes/movies"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_entity_type.ingest_from_df(\n",
    "    feature_ids=[\"average_rating\", \"title\", \"genres\"],\n",
    "    feature_time=\"update_time\",\n",
    "    df_source=backfill_movies_df,\n",
    "    entity_id_field=\"movie_id\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1e62Ku6W5OW_"
   },
   "source": [
    "#### Latest ingestion with imputed missing data\n",
    "Batch Serve from the latest ingestion with backfill/correction has reduced missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "3njvLv2wbZok"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Serving Featurestore feature values: projects/816732343711/locations/us-central1/featurestores/movie_predictions\n",
      "Serve Featurestore feature values backing LRO: projects/816732343711/locations/us-central1/featurestores/movie_predictions/operations/3882908064902283264\n",
      "Featurestore feature values served. Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions\n",
      "                  timestamp entity_type_users   age  gender     liked_genres  \\\n",
      "0 2021-09-15 08:28:14+00:00               dav   NaN    None               []   \n",
      "1 2021-09-15 08:28:14+00:00               eve  26.0    None         [Horror]   \n",
      "2 2020-02-14 09:35:15+00:00               bob  34.0    Male          [Drama]   \n",
      "3 2021-09-15 08:28:14+00:00               bob  35.0    Male  [Action, Crime]   \n",
      "4 2021-09-14 09:35:15+00:00             alice  55.0  Female          [Drama]   \n",
      "5 2021-09-15 08:28:14+00:00             alice  55.0  Female          [Drama]   \n",
      "\n",
      "  entity_type_movies                     title  average_rating   genres  \n",
      "0           movie_03           Cinema Paradiso             4.5  Romance  \n",
      "1           movie_04           The Dark Knight             4.6   Action  \n",
      "2           movie_04           The Dark Knight             4.2   Action  \n",
      "3           movie_02               The Shining             4.2   Horror  \n",
      "4           movie_03           Cinema Paradiso             4.5  Romance  \n",
      "5           movie_01  The Shawshank Redemption             4.9    Drama  \n"
     ]
    }
   ],
   "source": [
    "backfill_movie_predictions_df = movie_predictions_feature_store.batch_serve_to_df(\n",
    "    serving_feature_ids={\n",
    "        \"users\": [\"age\", \"gender\", \"liked_genres\"],\n",
    "        \"movies\": [\"title\", \"average_rating\", \"genres\"],\n",
    "    },\n",
    "    read_instances_df=read_instances_df,\n",
    ")\n",
    "print(backfill_movie_predictions_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpV-iwP9qw9c"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "You can also keep the project but delete the featurestore by running the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "NBTNfN8vxz4x"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Featurestore : projects/816732343711/locations/us-central1/featurestores/movie_predictions\n",
      "Delete Featurestore  backing LRO: projects/816732343711/locations/us-central1/operations/1667699998189420544\n",
      "Featurestore deleted. . Resource name: projects/816732343711/locations/us-central1/featurestores/movie_predictions\n"
     ]
    }
   ],
   "source": [
    "movie_predictions_feature_store.delete(force=True)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "sdk-feature-store-pandas.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m95",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m95"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
